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FOREWORD 


This  report  is  o^^e  of  a  series  of  reports  related  to  human  factors 
testing  in  Air  Force  Systems  and  represents  a  portion  of  the  applied 
research  program  of  the  Personnel  and  Training  Requirements 
Branch,  Training  Research  Division  of  the  Behavioral  Sciences 
Laboratory.  The  program  is  under  Project  1710,  "Training, 
Personnel  and  Psychological  Stress  Aspects  of  Bioastronautics," 
Task  171006,  "Personnel,  Training  and  Manning  Factors  in  the 
Conception  and  Design  of  Aerospace  Systems.  "  Dr.  Gordon  A. 
Eckstrand  was  the  project  scientist,  and  Mr.  Melvin  T.  Snyder 
was  the  task  scientist.  The  portion  of  the  work  reported  here 
was  sponsored  by  the  Training  Research  Division  through  the 
AFIT  graduate  program.  The  work  was  performed  by  Captain 
Stackfleth  in  1963  while  a  graduate  student  under  the  guidance 
of  Dr.  Ernest  J.  McCormick  at  Purdue  University. 

The  Behavioral  Sciences  Laboratory  has  initiated  a  program 
to  develop  methods  for  human  factors  testing  of  Air  Force 
Systems.  This  program,  in  support  of  personnel  subsystem 
test  and  evaluation,  has  at  least  two  facets:  The  evaluation  of 
human  performance  during  Category  I,  II  and  III  system  testing; 
and  the  evaluation  of  previously  developed  personnel  subsystem 
elements ,  like  Qualitative  and  Quantitative  Personnel  Require¬ 
ments  Information  (QQPRI).  This  report  is  related  to  the  second 
facet.  The  criterion  and  the  predictor  problems  cited  for  QQPRI 
are  probably  common  also  to  the  test  and  evaluation  of  other 
personnel  subsystem  elements. 

This  technical  report  has  been  reviewed  and  is  approved. 


WALTER  F .  GRETHER 
Technical  Director 
Behavioral  Sciences  Laboratory 
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ABSTRACT 


Some  of  the  problems  in  the  validation  of  personnel  requirements 
developed  and  predicted  in  the  Qualitative  and  Quantitative 
Personnel  Requirements  Information  reports  are  described. 
Included  are  problems  inherent  in  the  validation  procedures ,  such 
as  the  nature  of  the  predictor  (QQPRI) ,  the  problem  of  criterion 
selection  and  bias,  and  the  changing  nature  of  the  criterion. 
Because  of  the  multiple  nature  of  these  problems,  available 
testing  techniques  are  not  adequate  to  handle  the  testing  and 
provide  desired  information.  A  solution  is  presented.  This 
solution  requires  a  procedural  change  whereby  validations  are 
conducted  during  different  but  specific  stages  of  system 
development  and  test.  The  validations  would  be  oriented  to 
obtaining  the  best  validation  at  a  particular  time  and  for  a 
particular  purpose  rather  than  attempting  an  overall  test. 

Methods  are  suggested  for  determining  manning  deficiencies 
and  readjusting  the  personnel  subsystem. 
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INTRODUCTION 


Prior  to  the  weapon  system  concept,  the  individual  systems  of  an  aircraft  were 
assembled  from  state-of-the-art  equipment.  Components  and  the  accompanying 
skills  could  be  interchanged  easily  among  systems.  Sometimes  an  aircraft  was 
built,  but  operational  use  delayed  because  the  personnel  system  had  not  been 
provided  the  required  planning  factors.  As  a  result,  appropriate  decisions  were  not 
made  as  to  the  kinds  and  numbers  of  people  needed  to  operate  and  maintain  the  new 
weapon  system. 

Since  1954,  increasing  attention  has  been  focused  on  the  problem  of  placing 
adequately  trained  operator  and  maintenance  support  personnel  in  the  field  simul¬ 
taneously  with  the  hardware  and  procedures.  Initially,  many  organizations  and 
individuals  supported  work  in  the  areas  of  human  engineering,  personnel  require¬ 
ments  ,  and  training  requirements  to  meet  the  needs  of  these  weapon  systems .  As 
experience  was  gained,  it  became  increasingly  evident  that  these  functions  should 
not  be  separate  but  rather  that  all  aspectswhich  unite  the  human  component  with  the 
weapon  system  should  be  integrated  more  closely.  This  concept  resulted  in  an 
intra-Air-Force-command  agreement  designated  the  personnel  subsystem  (PS).  The 
PS  as  conceived  contains  eight  separate  elements,  one  of  which  is  PS  test  and 
evaluation.  (See  Appendix  I  for  list  of  PS  elements.) 

To  date,  possible  m.ethods  of  verifying  the  planning  that  has  gone  into  the  develop¬ 
ment  of  the  PS  for  any  given  weapon  through  PS  test  and  evaluation  are  almost  non¬ 
existent.  While  the  problem  of  testing  the  entire  PS  is  beyond  the  scope  of  this 
paper,  an  attempt  will  be  made  to  explore  some  of  the  problems  of  testing  on  PS 
element,  the  Qualitative  and  Quantitative  Personnel  Requirements  Information 
(QQPRI),  in  an  attempt  to  offer  suggestions  for  resolving  the  problem.  Until  these 
problems  are  stated  and  considered,  the  process  of  developing  more  detailed  test 
design  which  will  actually  provide  the  desired  verification  cannot  be  expected 
to  be  fruitful. 
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TEST  PURPOSES 


The  Handbook  of  Instructions  for  Aerospace  Personnel  Subsystem  Designers 
(HIAPSD)  (AFSCM  80-3)  states  that  there  are,  in  general,  two  primary  reasons  for 
testing  the  elements  of  the  personnel  subsystem.  These  are: 

1 .  To  determine  the  degree  to  which  the  PS  elements  meet  their 
objectives . 

2 .  To  point  out  the  nature  and  extent  of  deficiencies  which  may  exist 
in  any  of  the  given  PS  elements . 


Human  Performance  Test 

Shapero,  Cooper,  Rappaport,  and  Schaeffer  (ref  13)  while  evaluating  nine  Air  Force 
missiles,  reported  that  little  if  any  systematic  human  factors  performance  testing  is 
being  undertaken.  They  stated  that  an  analysis  of  4,248  malfunctions  indicated 
that  the  proportion  of  human  error  varied  from  20-53%  of  reported  equipment  failures, 
and  from  16-23%  of  the  unscheduled  delays.  Even  so,  the  malfunction  data  collec¬ 
tion  system  being  utilized  was  inadequate  for  identifying  or  obtaining  pertinent  data 
on  human  initiated  equipment  malfunctions . 

Because  of  the  influence  of  the  human  component  on  malfunctions,  Shapero  suggests 
that  for  human  engineering,  some  method  of  test  should  be  established  which  will 
meet  the  following  criteria  where  human  operators  are  concerned; 

1.  There  should  be  a  method  for  obtaining  consistently  defensible, 
observable  measures  about  the  human  operator  with  procedures 
for  validating  these  observations. 

2.  These  methods  should  be  referable  to  existing  models  of  systems  of 
known  components. 

3.  These  methods  should  allow  modification  of  work  areas  and  work 
places . 

4.  They  should  account  for  where  and  how  the  human  operator  enters 
the  system  at  each  level  of  the  system. 


Personnel  Requirements 

Demaree,  Marks,  Smith,  and  Snyder  (ref  4)  indicate  that  three  specific  purposes 
are  served  by  the  QQPRI  program; 
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1.  Identification  of  Air  Force  specialties  and  specialty  codes  for 
the  positions  required  for  maintenance  and  operation  of  a  given 
Air  Force  system . 

2.  Development  of  essential  information  for  the  organizational 
tables  and  unit  manning  documents . 

3.  Provision  of  timely  and  valid  information  for  the  planning  of 
training  courses  and  other  system  supports. 

If  these  are  the  purposes  of  QQPRI,  then  the  problem  seems  to  be  one  of  testing  how 
accurately  the  prediction  of  Air  Force  specialties,  numbers  of  personnel  required, 
and  amount  of  training  required  have  been  accomplished.  If  the  prediction  and 
criterion  measures  are  in  accord,  no  problem  exists.  Powe,  Carrier,  and  Skandera 
(ref  12)  while  conducting  a  test  of  the  human  factors  portion  of  the  IM  99  A  system 
reported  an  inadequate  number  of  personnel  available  to  process  the  missile  and  an 
inadequate  percentage  of  7  level  (skilled)  airmen .  This  test  indicated  that  the 
QQPRI  grossly  underestimated  the  requirements  in  these  two  areas.  What  were 
some  of  the  reasons  for  this  underestimation?  For  one  thing,  none  of  the  automatic 
checkout  equipment  operated  in  a  fully  automatic  manner ,  and  continuous  contractor 
assistance  was  required  with  "work  around"  procedures  being  developed  for  the 
operation.  In  any  case,  the  operational  plan  could  not  be  met  and  modification  of 
the  facilities  was  required.  Thus,  we  have  an  indication  that,  while  the  personnel 
requirements  were  in  error,  this  was  in  part  a  result  of  factors  concerned  v/ith  the 
weapon  system  as  a  whole  as  well  as  the  personnel  system.  Losee,  Payfer,  Frahm, 
and  Eisenberg  (ref  9)  have  shown  that  manpower  requirements  permeate  every  aspect 
of  the  weapon  system.  Thus,  the  sources  of  error  possible  are  dependent  not  only 
upon  the  direct  aspects  of  the  personnel  prediction,  but  also  any  facet  of  the 
weapon  which  might  constitute  an  error  source  (see  figure  1) . 


Figure  1 


Interrelations  of  Personnel  and  System 
Information  (Adapted  from  Losee  et  al, 
ref  9 ,  page  8) 
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Errors  in  any  of  the  information  sources  feeding  into  the  QQPRI  will  have  an  impact 
on  the  final  personnel  predictions  contained  in  the  QQPRI,  and  while  not  illustrated 
in  figure  1,  the  QQPRI  may  in  turn  modify  the  assumptions  concerning  the  complete 
weapon  system. 

dynamic  nature  of  the  predictor 

OOPRI  Revision 

When  considering  the  validation  of  the  QQPRI,  a  somewhat  different  situation  exists 
from  the  normal  test  design.  For  one  thing,  since  QQPRI  is  essentially  a  planning 
program  and  document,  the  knowledge  of  the  predictor  is  widespread,  and  by 
definition  every  attempt  is  made  to  correlate  the  predictor  with  the  "criterion, " 
so  that  right  or  wrong,  once  the  prediction  has  been  made,  the  intent  is  that  the 
predictor  shall  become  fact.  In  those  cases  where  operational  or  hardware  modifica¬ 
tions  are  implemented,  the  QQPRI  is  also  modified.  Losee  et  al,  ref  9,  p  2, stated: 

A  static  QQPRI  would  reflect  only  the  original  design  thinking  and 
the  original  concepts.  Since  development  of  aerospace  craft, 
missile,  or  electronic  weapon  systems  must  necessarily  be 
dynamic  in  nature,  the  QQPRI  must  also  be  dynamic,  ie,  an 
updating  process  must  occur  .... 

Prediction  at  Various  Stages  of 
Weapon  System  Development 

Losee  et  al  further  suggest  that  such  QQPRI  revisions  might  reasonably  be 
expected  at  the  time  of  the  first  firm  drawing  release,  at  mock-up  tim.e,  during 
prototype  testing,  following  Category  II  test,  and  during  the  production  phases. 
These  revisions  may  or  may  not  result  in  discrete  publications  depending  upon  the 
extent  of  the  changes.  On  the  basis  of  a  study  by  Gianzer  and  Glaser  (ref  7) , 

Losee  and  others  have  recommended  that  the  best  prediction  can  be  made  following 
development  of  prototype  equipment.  This  may  be  the  earliest  prediction  which 
can  be  made  with  reasonable  stability.  However,  in  view  of  the  dynamics  of  the 
situation,  accuracy  of  the  prediction  may  well  be  inversely  proportional  to  the 
length  of  time  between  the  prediction  and  testing ,  or  more  specifically  to  the 
degree  of  cha:  ge  occurring  in  the  system  between  prototype  development  and  the 
final  operational  configuration.  This  is  based  upon  the  assumption  that  as  additional 
information  becomes  available  and  designs  are  firmed  up,  fewer  changes  will  occur 
between  the  prediction  and  the  test  resulting  in  greater  validity.  It  would  be  a 
dangerous  assumption  to  infer  that  because  an  item  of  equipment  had  reached 
some  milestone  in  development  and  was  successfully  tested  at  that  point,  that  other 
systems  would  give  similar  test  results  at  that  point  in  time. 
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Ghiselli  (ref  6)  in  discussing  dimensional  problems  of  criteria,  points  out  that  in  a 
particular  testing  situation,  he  found  that  the  point  in  time  that  a  criterion  measure 
was  observed  influenced  the  accuracy  of  his  prediction.  He  concluded  that  to  obtain 
validity,  prediction  would  have  to  be  of  performance  fairly  closely  pinpointed  in 
time. 

Because  of  the  dynamic  nature  of  the  predictor,  the  decision  as  to  when  the  valida¬ 
tion  should  take  place  will,  in  general,  be  a  major  factor  with  regard  to  how  weli 
the  predictor  relates  to  the  criterion,  since  I  would  expect  that  the  closer  in  time 
the  two  occur,  the  better  will  be  the  prediction. 

DYNAMIC  NATURE  OF  THE  CRITERION 


A  problem  of  major  concern  is  the  point  in  time  during  which  validation  of  personnel 
requirements  should  occur.  The  earlier  the  test  can  be  made,  the  more  opportunity 
for  correcting  deficiencies  before  the  weapon  system  is  introduced  into  the  field 
with  its  attendant  costs  of  trained  personnel .  The  earlier  required  modifications 
are  made,  the  less  costly  subsequent  planning  operations  will  be.  An  initial 
estimate  might  suggest  that  this  evaluation  should  be  accomplished  during  weapon 
system  test. 


Weapon  SystemjTest 

A  period  of  time  (Category  I,  II,  and  III  tests)*  is  allotted  to  each  weapon  system 
for  testing  before  operational  use.  However,  several  aspects  of  the  weapon  system 
test  situation  do  not  lend  themselves  to  validation  of  the  predicted  personnel 
requirements  of  the  system: 

1.  System  design  may  vary  during  the  test  situation.  For  example, 
ground  support  equipment  may  not  be  developed  so  that  it  can 
be  used  in  conjunction  with  the  prime  equipment.  The  use  of 
substitute  equipment  may  drastically  change  the  duties  and  tasks 
which  are  to  be  performed  during  the  test  period.  Likev/ise, 
equipment  reliability  at  this  time  may  not  be  representative  of 
the  reliability  which  can  be  expected  after  operational  use,  so 
that  the  frequency  of  duty  and  task  performance  is  likewise  not 
representative.  The  few  available  items  of  equipment  are  being 
fully  utilized  for  hardware  test,  so  they  are  difficult  to  obtain 
for  testing  the  human  component. 

2.  Personnel  who  actually  participate  in  the  test  may  not  be  similar 
in  skills  to  the  operational  personnel.  During  the  early  phases 


*See  Appendix  H  for  description  of  Category  I,  II.  and  III  tests. 
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of  testing,  equipment  test  is  handled  by  contractor  personnel 
already  stationed  at  the  test  site.  During  Category  III  tests, 
the  individuals  sent  from  the  operational  squadrons  may  not 
be  nepresentative  in  skill  or  knowledge  to  those  airmen  who 
will  eventually  he  expected,  to  operat  e  the  system  In  the 
operational  contex*. 

3.  Learning  factors  can  be  expected  to  increase  the  time  required  in 
performing  jobs  both  in  the  test  and  early  operational  phases  of 
the  system. 

Testing  the  personnel  prediction  at  this  point  in  time  results  in  a  test  conducted 
under  different  assumptions  than  those  for  which  the  prediction  was  developed. 

That  appropriate  personnel  must  be  available  to  support  system  test  may  be  a  valid 
argument,  but  to  do  this  the  underlying  assumptions  pertaining  to  personnel  require¬ 
ments  must  be  modified  to  conform  to  the  testing  situation. 

Brogden  and  Taylor  (ref  1)  have  suggested  four  biasing  factors  that  may  produce  a 
deviation  of  obtained  criterion  scores  from  a  hypothetical  "true"  criterion  score. 
These  factors  are  classified  as: 

1 .  Criterion  deficiency — omission  of  pertinent  elements  from  the 
criterion . 

2.  Criterion  contamination — introducing  extraneous  elements  into 
the  criterion. 

3 .  Criterion  distortion — improper  weighting  in  combining  criterion 
elements . 

4.  Criterion  scale  unit  bias — inequality  of  scale  units  in  the 
criterion . 

The  first  three  items  are  particularly  relevant  to  testing  QQPRI  during  weapon 
system  test. 


Criterion  Selection 

Another  problem  to  be  considered  is  the  difficulty  involved  in  selecting  and 
establishing  a  criterion.  Even  if  the  criteria  could  be  stated  simply  as  "the 
number  and  types  of  personnel  required  to  actually  operate  the  system  in  the  field, " 
it  would  still  be  difficult  to  obtain  an  accurate  picture  of  just  how  many  personnel 
are  required.*  What  approach  might  be  used  to  answer  such  a  question?  Certainly 


*The  problem  of  overall  manpower  utilization  appears  to  be  a  shade  different  than 
performance  testing  of  individuals  in  their  respective  jobs. 
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at  present  insufficient  data  exists  regarding  what  a  man  can  do,  or  what  constitutes 
a  full  day's  work,  so  that  an  answer  may  reasonably  be  expected  to  come  as  a 
result  of  "expert"  judgment.  Who  is  in  the  best  position  to  judge  the  field  situa¬ 
tion  ?  The  unit  commander  and  his  staff  are  logical  candidates ,  however  they  may 
be  somewhat  biased  toward  increasing  the  number  of  personnel  required.  This 
does  not  necessarily  result  from  Parkinson's  Law,  but  comes  from  the  very  real  and 
practical  situation  of  having  suffered  through  personnel  acts  during  previous  periods 
so  that  a  little  "padding"  might  be  regarded  as  genuine  foresight. 

An  alternative  method  might  utilize  the  judgment  of  personnel  knowledgeable  about 
the  problems  and  operations  of  several  similar  bases  or  commands.  Having  access 
to  a  larger  scope  of  operations  via  observation  of  similar  organizations  might  oe  a 
reasonable  place  to  obtain  judgments  of  what  is  actually  a  fair  proportion  of  the 
available  manpower  which  can  be  allotted  to  a  functional  unit.  This  too  has  the 
possible  disadvantage  of  contamination  of  criteria  because  here  the  inclination  is 
toward  a  proportionate  share  of  manpower  and  skills  rather  than  a  judgment  of  what 
is  actually  dictated  by  the  job.  A  combined  judgment  similar  to  that  used  during 
the  development  of  personnel  prediction  information  might  assist  in  this  phase; 
that  is ,  utilizing  a  combination  of  unit  personnel  and  higher  echelon  personnel 
or  an  independent  group  of  well  qualified  judges . 

Clifton  (ref  3)  in  a  study  of  staffing  requirements  for  Navy  CVA  59  and  61  class  ships 
found  reliabilities  generally  in  the  .90's  among  estimates  of  desirable  and 
minimum  levels  of  personnel,  as  compared  to  numbers  of  personnel  allocated  by  the 
Bureau  of  Naval  Personnel.  While  this  situation  differs  somewhat  in  that  the  ships 
were  already  in  commission,  the  stability  of  judgments  does  provide  an  indication 
that  usable  criteria  can  be  obtained. 

Assuming  that  simple  criteria  can  be  established  for  validation  purposes,  will  a 
simple  criterion  be  sufficient? 

Need  for  Multiple  Criteria 

Noriega,  Jowdy,  and  Palmer  (ref  11)  in  developing  a  methodology  for  optimum 
manning  for  base  level  aircraft  maintenance  have  suggested  several  criteria  which 
they  feel  are  important  fora  personnel  prediction  system.  A  few  of  these  are  as 
follows: 

1.  Maintenance  manning  should  be  oriented  toward  operational 
readiness . 

2.  Optimal  utilization  of  USAF  maintenance  manpower  should  be  provided. 

3.  Maintenance  support  requirements  of  operational  units  must  be  met. 

4.  Objective  justification  and  validation  of  manpower  requirements 
must  be  provided. 
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5.  The  prediction  system  must  be  usable  within  the  parameters  of 
current  organizational  operations  and  maintenance  concepts  and 
be  responsive  to  subsequent  changes  in  these  concept-. 

6.  The  prediction  system  must  be  responsive  to  all  factors  influ¬ 
encing  maintenance  workload . 

7 .  Portions  of  total  workload  by  quantity ,  quality ,  and  types  of 
skill  categories  must  be  isolated. 

8.  AFM  26-1  must  be  utilized. 

9.  Means  of  identifying  manning  expenditures  above  and  below 
established  requirements  must  be  provided. 

Review  of  these  items  indicates  two  important  factors;  first,  that  a  single  criterion 
would  not  provide  a  sufficient  basis  for  validation,  and  second,  a  mixture  of 
criteria  has  occurred,  one  facet  dealing  with  criteria  for  the  predictor  (5)  and  the 
other  pertaining  to  some  measure  of  field  test  (3) .  From  the  viewpoint  of  ultimate 
requirements  both  facets  are  required,  but  the  question  remains,  are  they  com¬ 
patible  for  a  single  test? 

Validation  Using  a  "Best”  Predictor  and 
"Best”  Criterion 

One  approach  would  be  to  select  a  "best"  point  in  time  for  the  predictor  to  be 
held  constant,  then  establish  a  criterion  during  that  point 'in  time  when  the  system 
is  considered  to  be  in  its  average  operational  condition,  that  is,  subsequent  to  the 
time  when  the  majority  of  the  "bugs"  have  been  eliminated  from  the  system  and 
the  personnel  maintaining  it  have  becoir  familiar  with  the  equipment.  By  comparing 
the  predictor  with  the  criterion  at  these  ^elected  points  in  time,  it  would  be 
possible  to  determine  the  degree  of  accuracy  with  which  predictions  were  made  on 
that  system.  Unfortunately,  this  "best"  point  in  time  would  probably  not  occur 
until  the  system  was  already  operational.  The  knowledge  gained  at  this  stage 
would  be  of  little  help  for  the  system  under  study.  However,  it  would  have  the 
advantage  of  recording  the  deficiencies  of  a  system  presently  in  operation,  then 
insuring  that  procedures  for  procuring  that  type  of  data  were  incorporated  into 
future  prediction  procedures.  Because  of  the  number  of  variables  involved  and  the 
changing  conditions  resulting  from  a  lapse  of  years  between  prediction  and  valida¬ 
tion,  the  information  received  would  at  best  be  of  a  general  rather  than  a  specific 
form. 


Mid— Point  Resume 

In  view  of  the  diverse  problems  discussed  up  to  this  point,  perhaps  a  short  sum¬ 
mation  is  in  order: 
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1.  It  is  necessary  that  some  evaluation  of  personnel  requirements 
be  made  at  an  early  time  period . 

2.  Many  factors  aside  from  the  personnel  prediction  will  influence 
the  results  of  these  evaluations.  Among  these  are: 

a.  The  inseparability  of  the  equipment  and  personnel  factors. 

b.  The  time  chosen  for  conducting  the  test. 

c.  The  decisions  regarding  the  constituents  of  the 
ultimate  criterion. 

3.  By  holding  several  factors  constant,  a  test  could  be  run  on  a 
specific  system  to  determine  deficiencies  in  the  prediction  system. 
However,  the  information  would  be  expected  to  be  general  in 
nature,  too  late  for  major  contribution  to  the  weapon  system 
being  tested,  and  not  generalizable  to  a  future  system. 

4.  In  view  of  these  factors,  it  does  not  appear  practical  to  test 
simultaneously  the  adequacy  of  the  weapon  system  and  the 
adequacy  of  the  predictor. 

AN  ALTERNATIVE  APPROACH  TO  VALIDATION 

Chase  (ref  2)  has  suggested  that  two  phases  exist  in  the  validation  problem: 

1.  Testing  the  basic  data  from  which  predictor  information  is  later 
developed. 

2.  Evaluation  of  actual  performance  on  the  job:  that  is,  whether  or 
not  the  personnel  can  perform  their  mission.  He  suggests  that 
this  type  of  validation  can  be  performed  through  observation  of 
personnel  performance . 

Since  studies  of  the  evaluation  of  performance  have  already  been  undertaken,  a 
brief  review  of  this  topic  may  be  appropriate  before  proceeding  to  a  discussion  of 
validating  the  basic  data. 
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Human  Performance  Evaluation 


The  procedures  and  techniques  used  in  human  performance  evaluation  are  by  no 
means  new,  although  their  application  in  the  weapon  system  test  program  has  been 
somewhat  limited.  Aside  from  classified  tests,  examples  include: 

1.  Powe,  Carrier,  and  Skandera  (ref  12)  used  checklists  and 
interviews  as  well  as  observation  during  human  factors  test 
of  the  IM  99  Weapon  System. 

2 .  Clifton  (ref  3)  in  attempting  to  determine  staffing  requirements 
for  naval  vessels,  used  a  diary  approach  with  good  result. 

3.  Marx  (ref  10)  recently  considered  in  some  detail  the  methods 
of  evaluating  human  performance.  This  report  thus  serves  as 
an  additional  source  of  information  on  these  procedures . 

In  general,  application  of  techniques  contained  in  AFM  35-1  and  AFM  26-1  may 
also  furnish  appropriate  guidelines  for  evaluating  the  numbers  and  types  of 
personnel  in  those  instances  where  incumbents  have  actual  job  experience.  Where 
problem  areas  are  encountered  which  result  from  a  difference  between  the  actual 
operational  situation  and  the  QQPRI,  updating  the  QQPRI  may  serve  as  a  guide  to 
revising  personnel  requirements.  For  example,  if  sufficient  personnel  are  not 
available,  and  the  malfunction  rate  is  higher  than  predicted,  substituting  a  revised 
frequency  of  task  performance  may  indicate  whether  this  factor  alone  resolved  the 
problem  or  whether  other  factors  were  also  contributing. 

In  any  event,  while  the  predictor  information  is  not  in  itself  being  tested,  the 
techniques  and  information  can  be  useful  during  performance  test  as  cited  above 
or  in  pinpointing  specific  test  areas  which  were  expected  to  present  problems  such 
as:  critical  tasks ,  safety  hazards ,  complex  training,  etc. 

Validating  Basic  Prediction  Data 

Checking  the  basic  data  may  cover  several  stages.  First,  an  initial  check  of  the 
information  will  be  made  by  the  Air  Force  as  it  is  received  from  the  contractor. 

One  of  the  possibilities  here  is  to  make  checks  individually  of  the  logical  points 
where  omissions  of  important  aspects  of  the  system  may  occur.  Because  several 
hundred  tasks  may  be  listed  with  each  being  rated  according  to  such  characteristics 
as  criticality,  newness,  complexity,  etc,  a  complete  recheck  may  not  be  feasible 
in  the  time  available  for  review.  To  accelerate  this  process,  one  could  reasonably 
make  spot  checks. 

A  matrix  (see  figure  2)  might  provide  a  convenient  reference  for  illustrating  com.bina- 
tions  of  task  statements  that  are  not  entirely  consistent  or  that  might  indicate  tasks 
requiring  special  attention.  For  example,  if  a  task  is  rated  as  requiring  an 
extended  period  of  training  time  and  is  also  checked  as  being  critical,  then  further 
checking  of  the  whole  task  may  be  worthwhile.  Likewise,  a  task  that  has  new 
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Aerospace  Ground  Equipment  (AGE)  and  requires  extended  training  should  receive 
additional  attention.  Such  a  procedure  could,  using  a  card  sorting  system, 
automatically  sort  out  combinations  of  task  characteristics  which  indicate  either 
the  more  important  tasks  or  those  which  present  inconsistencies. 


Figure  2.  Matrix  for  Task  Qualities 


NOTE:  NAT  is  No  Additional  Training;  OJT  is  On  Job 
Training;  EFT  is  Extended  Formal  Training 


As  a  further  example,  if  in  sorting  the  cards,  an  item  of  new  AGE  was  found  to  be 
included  in  the  position  but  the  training  description  was  marked.  No  Additional 
Training  (NAT)  ,  then  the  analyst  must  be  satisfied  that  the  new  item  of  equipment 
did  not  in  reality  modify  the  tasks  in  that  position.  An  unusually  large  frequency 
of  errors  in  any  one  position  would  indicate  that  a  more  detailed  check  of  that 
particular  position  was  required. 

Other  techniques  at  this  stage  would  also  be  helpful.  For  instance,  check  lists 
could  be  developed  which  would  aid  the  analyst  in  reviewing  the  report  prior  to 
publication 


11 


AMRL-TDR-64-65 


Checkiisfs 


Included  in  such  a  checklist  would  be  items  such  as: 

1 .  Do  the  number  of  aircraft  programmed  for  maintenance  and  the 
number  allocated  to  training  flights,  etc,  add  up  to  the  total 
number  of  aircraft  assigned  to  the  squadron? 

2.  Have  appropriate  allowances  (as  per  Air  Force  Manuals)  been 
made  for  leave,  sickness,  etc,  when  considering  personnel 
available  for  duty? 

3o  Recheck  other  clerical  verifications  already  built  into  the 
prediction  form. 


Task  Simulation 

Vacherot  and  Teeple  (ref  14)  have  proposed  a  system  -for  simulation  of  manpower 
requirements.  The  methodology  of  that  system  appears  to  present  some  rather 
interesting  prospects  of  being  adaptable  not  only  for  prediction  purposes ,  but 
perhaps  for  use  in  integrating  several  techniques  developed  independently.  This 
technique  involves  the  simulation  of  a  24-hour  period  reduced  to  1-1/2  hours  by 
using  digital  computers.  In  this  scheme,  factors  such  as  the  following  are 
included: 

1.  Location  and  time  at  which  the  task  commences. 

2.  The  location,  number,  and  capabilities  of  men  available  to 
perform  the  task. 

3.  The  logical  rules  to  be  employed  in  treating  tasks  and  men. 


The  program  commences  with  a  number  of  men  with  specified  abilities  at  a  given 
location.  Then  the  tasks  to  be  performed  are  introduced  chronologically.  This 
procedure  continues  on  a  24-hour  basis  to  measure  the  efficiency  of  manpower 
utilization.  The  procedure  is  divided  into  three  phases: 

1 .  The  load  generation  phase  in  which  the  task  frequencies  for 
24  hours  are  programmed. 


2.  The  originating  time  of  the  task  is  programmed. 

3.  A  task  length  is  chosen  at  random  from  a  specified  distribution. 
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The  AFSC  priorities  are  established,  the  borrowing  priorities  are  established,  and 
the  maximum  delay  time  is  fixed , 

During  the  simulation  phase,  the  tasks  are  fed  into  the  process  on  the  basis  of  the 
programmed  information.  If  the  task  can  be  delayed,  it  is  and  if  it  is  not 
delayable,  another  tasks  is  substituted.  Then  the  priorities  for  bon-ov\ring  are 
searched  and  other  personnel  are  used  with  consideration  being  given  to  their 
non-availability  for  the  performance  of  previously  assigned  tasks.  If  the  search  is 
not  fruitful,  then  the  requirement  for  the  "creation"  of  a  new  position  exists. 

The  methodology  proposed  by  Losee  et  al  has  attempted  to  show  how  a  great  number 
of  the  variables  affecting  personnel  requirements  can  be  integrated  for  a  single 
weapon  system.  The  contents  of  their  report  consider  in  some  detail  the  importance 
of  task  frequencies  in  determining  manning  requirements,  and  present  some  new 
methods  for  estimating  the  frequencies  from  equipment  reliabilities.  Other  phases 
proposed  in  this  simulation  model  would  also  be  available  as  an  outgrowth  of 
using  the  technique  proposed  by  Losee.  Examples  are: 

1.  Availability  of  task  information  in  sequence. 

2 .  Length  of  tasks . 

3.  Delays  resulting  from  differences  in  task  location. 

Thus ,  the  personnel  requirements  predictions  would  be  established  initially  utilizing 
the  basic  assumptions,  task  estimates,  and  other  data  which  could  be  made  available 
on  the  basis  of  the  requirements  generated  by  the  weapon  system.  Gael  and 
Stackfleth  (ref  5)  have  shown  the  feasibility  of  using  card  systems  for  the  initial 
data  collection  for  this  purpose,  and  a  card  system  was  used  for  data  collection  in 
the  Minuteman  Missile  program.  The  manpower  prediction  technique  proposed  by 
Losee  was  designed  to  be  adaptable  to  card  or  tape  programming  techniques  which 
would  allow  not  only  the  fallout  of  the  original  data  for  planning  information,  but 
would,  with  minor  modifications,  allow  the  simulation  procedure  to  utilize  the  same 
data  for  a  more  thorough  and  rapid  check  than  has  been  possible  in  the  past.  Thus, 
within  a  short  time  span  the  system  cycle  could  be  completed.  At  this  stage  it  would 
be  known  whether  the  system  as  conceived  and  planned  could  operate  under  the 
known  conditions  imposed  upon  it.  The  degree  of  latitude  in  terms  of  assumptions 
and  estimates  would  be  more  fully  known,  and  in  addition,  this  technique  would 
allow  research  on  those  factors  tending  to  affect  the  final  results  in  terms  of 
numbers  of  people  required,  thus  permitting  additional  insights  into  those  areas 
which  should  be  looked  at  more  closely  during  weapon  system  test. 

The  probabilities  of  malfunctions  on  which  the  task  frequencies  are  based  should  be 
very  amenable  to  this  kind  of  programming.  Additional  research  in  the  area  of 
programming  skills  and  knowledges  would  be  required. 
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The  first  run  of  the  program  would  essentially  provide  data  regarding  deficiencies 
of  the  prediction  system.  Any  instance  where  a  problem  was  encountered  would  be 
recorded  and  subsequently  analyzed  for  improvement  of  later  predictions,  and  any 
required  changes  in  the  prediction  system  could  be  corrected  to  remove  these 
deficiencies.  In  essence  then,  this  simulation  would  be  validating  the  prediction 
system.  On  subsequent  runs,  ie,  after  the  first,  the  simulation  technique  could 
be  used  to  better  design  the  personnel  area  in  terms  of  its  own  internal  consistency. 


SUMMARY  AND  CONCLUSIONS 


Because  of  the  continuing  requirement  for  valid  personnel  information  pertaining  to 
new  weapon  systems ,  a  requirement  exists  for  testing  and  evaluating  prediction  of 
personnel  requirements .  This  report  reviews  some  of  the  problems  in  the  validation 
procedure,  such  as  the  nature  of  the  predictor  (QQPRI),  the  problem  of  criterion 
selection  and  bias,  and  the  changing  nature  of  the  criterion.  I  conclude  that, 
because  of  the  nature  of  this  problem,  available  testing  techniques  are  not 
sufficient  to  handle  the  multiple  problems  and  provide  the  desired  information; 
rather  it  is  suggested  that  the  validation  procedure  be  conducted  during  different 
stages  specifically  oriented  toward  obtaining  the  best  validation  at  a  particular 
time  and  for  a  particular  purpose  rather  than  attempting  an  overall  test.  Sugges¬ 
tions  are  included  for  performing  these  tests  to  obtain  the  desired  result.  In 
particular,  a  system  of  task  simulation  is  suggested  that  may  be  of  use  both  in 
determining  manning  deficiencies  and  in  redesigning  portions  of  the  Personnel 
Subsystem. 
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APPENDIX  I 

PERSONNEL  SUBSYSTEM  ELEMENTS 


1.  Personnel-Equipment  Data  (PED). 

2.  Human  Engineering  (HE). 

3.  Qualitative  and  Quantitative  Personnel  Requirements  Information  (QQPRI). 

4.  ;  Training  Plans  (TP). 

5.  Training  Equipment  Planning  Information  (TEPI). 

6.  Training  Equipment  Development  (TED). 

7.  I  Technical  Orders  and  Technical  Manuals  (TOTM's). 

8.  V  Personnel  Subsystem  Test  and  Evaluation  (PSTE) . 
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WEAPON  SYSTEMS  TESTS 


Category  I  —  Subsystem  Development  Tests  and  Evaluation 

This  consists  of  the  individual  components  and  subsystems  of  a  system.  In  addition 
to  qualification,  this  testing  provides  for  redesign,  refinement  and  reevaluation  as 
necessary.  Some  of  the  test  objectives  are  to  determine: 

1.  Performance,  reliability,  and  integrity  of  individual  components. 

2.  Compatibility  of  noted  components. 

3.  Compatibility  of  government  furnished  equipment  or  standard  Air  Force 
items  for  Incorporation  into  the  system. 

4.  Airborne  I  operation  characteristics  and  air  worthiness  of  subsystems  and 
components  of  the  system. 

5.,  Preliminary  performance,  stability  and  control  characteristics,  and 
general  air  worthiness  of  the  air  vehicle. 

6.  Compatibilit/  and  adequacy  of  ground  support  equipment. 

7.  Preliminary  maintainability  characteristics  of  components  and  subsystems . 

8.  Initial  requirements  for  .personnel  and  training,  skill  identification  and 
the  adequacy  of  personnel  training  devices. 


Category  It—  System  Deve.lopment  Test  and  Evaluation 

This  consists  of  development  testing  and  evaluation  of  integrated  subsystems.  It 
v/ill  Include  functional  and  development  tests  and  military  demonstration  of  the  whole 
system  in  as  realistic  and  complete  environment  as  practicable. 
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Some  of  the  test  objectives  are: 

1.  Determine  compliance  with  specifications  for  performance,  stability, 
control,  and  maintenance;  and  obtain  necessary  data  for  handbook 
and  other  publications . 

2.  Evaluate  new  design  changes  for  production. 

3.  Determine  capabilities  under  simulated  climatic  conditions. 

4.  Demonstrate  in  as  realistic  and  complete  environment  as  practicable  that 
the  whole  system  is  functionally  operative,  operationally  effective,  and 
compatible  with  the  other  systems  and  supporting  equipment  required  for 
its  operative  employment. 

5 .  Determine  if  the  system  is  capable  of  and  suitable  for  meeting  the 
established  requirements  and  design  objectives. 

6 .  Provide  opportunity  for  familiarization ,  experience ,  and  limited  training 
on  the  system  to  prime  major  air  command  and  ATC  personnel. 

7.  Demonstrate  in  the  most  realistic  environment  practicable  that  the 
complete  system  is  maintainable  with  minimum  resource  outlays ,  ie , 
support  and  test  equipment,  personnel  and  skills,  special  tools  and 
training,  spare  parts,  and  special  facilities. 

8.  Determine  the  adequacy  of  ground  support  equipment. 


Category  ill  —  System  Operational  Test  and  Evaluation 

This  consists  of  a  major  air  command  user  tests  and  evaluations  of  operationally 
configured  systems  with  all  components ,  support  items  and  personnel  skills ,  under 
operational  conditions.  Some  of  the  test  objectives  are: 

1 .  Determine  the  operational  usefulness  of  the  system  and  develop  the 
most  effective  operational  tactics,  techniques,  doctrines  and  standards. 

2 .  Determine  any  operational  deficiencies  and  provide  quantitative  and 
qualitative  data  for  product  improvement  programs . 

3.  Obtain  supplemental  data  on  the  rate  of  parts  consi  rnption,  maintenance 
and  support  facility  requirements  obtained  during  previous  tests. 

4.  Obtain  supplemental  datv=i  on  organizational  and  personnel  skill  and 
training  requirements  procured  during  previous  tests. 
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5.  Evaluate  the  adequacy  of  the  Table  of  Organization,  the  training  program, 
and  the  authorization  documents . 

6.  Obtain  supplemental  data  relative  to  minimum  maintenance  requirements  in 
terms  of  personnel,  skills,  and  training;  special  tools,  test  and  support 
equipment;  special  facilities:  general  performance  standards  for  doing 
maintenance  tasks. 

7.  Qualify  and  integrate  the  first  using  command  units  into  the  operational 
inventory . 
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